Binary Data Formats

Introduction#

Text-based data representation is often preferred when interacting with different clients, especially when client implementations are outside of an organization’s control. This is because it’s easier to analyze and debug when the data is human-readable. For internal use, such as in iOS and Android applications (where the organization controls both ends of the communication tightly), there’s more freedom and flexibility in choosing the data format, and we can move to formats that are more compact, faster to parse, and lighter on the database. The binary encoding formats share the following benefits in common:

• Machine-friendly: The data is in binary format and can be processed with little or no preprocessing.

• Schema dependent: Data-related information is defined in the schema document.

• Portability: Binary formats can be easily deserialized into different languages ​​as long as they have the same encoding and decoding schema.

• Precision support: Binary format can store large numbers (variable length integers) and floating point numbers with greater precision.

• Standardized: The binary format uses a well-defined schema that's well-documented and can be standardized across different implementations.

Many binary formats exist, such as MessagePack, Fast Infoset, BSON, WBXML, Protobuf, Thrift, Avro, and so on. Let's see how the message object above looks when encoded in MessagePack, a binary variant of JSON.

The binary version of the above message object is demonstrated in the image below:

MessagePack and other binary variants of JSON and XML follow a schemaless encoding style that increases the size of the encoded data and are not common in API development. For example, in the above illustration, disablelinks takes an extra 12 bytes to encode the field name, whereas its value is only one byte, representing true.

Thrift and Protobuf#

The most commonly used binary data formats are Protocol buffers (Protobuf) and Apache Thrift. Google developed Protocol Buffers, and Facebook developed Apache Thrift. Both these binary encoding libraries follow similar rules and have been open-sourced for standardized public use. They have a similar-looking schema, shown below:

We can breakdown these schema definitions as follows:

• Tags: Each field is identified by the numbers equivalent to the names. For example, 1 identifies the fields nodeID for Thrift and node_id for Protobuf. These tags are encoded in data rather than field names, allowing for more compact data representation.

• Labels: The markers required, optional, and repeated (Protobuf only) imply checks to determine whether to raise an error when encoding or decoding data. These markers are only added to the schema but are not encoded in the transmitted data.

• Types: Each field can define its data type (string, i64, int64, etc.) by specifying a type marker. Protobuf does not have a type to represent arrays, and it uses the repeated label to show that the data can have multiple occurrences.

• Names: The markers (nodeID, node_id, etc.) represent the names of the data entity. One thing worth noting here is that these are not identifiers for fields, which allow the name to be changed without affecting the implementation.

Note: The use of the label, repeated, in Protobuf allows flexibility to change a single variable to an array of variables in different schema versions, but it’s only valid for optional fields, otherwise it can cause compatibility problems. To read more, see Thrift documentation and Protobuf documentation.

Apache Avro#

Apache Avro is also an open-source binary encoding format developed by Hadoop. Avro supports two schema styles, Avro IDL and JSON-based definitions. JSON is well-known in the industry and allows developers to create schemas without having to learn another language. Avro doesn’t add tags and types like Thrift and Protobuf and uses a value-only data encoding style. It relies on the schema version defined in the data reader to identify the fields when decoded. If there are any conflicts due to different versions of the writer's and reader's schema, the Avro library resolves these conflicts through a side-by-side comparison of these versions, as shown below.

Here, the field name is used as the field's identifier, so the records can be in any order as long as the field name remains the same. If the reader code finds a field that is not in the reader schema, it just ignores that field. On the other hand, if the record doesn’t contain the fields expected by the reader schema, the reader code populates it with the default values ​​specified in the reader schema.

To read more on Apache Avro see the Apache Avro Documentation.

Avro is very flexible with dynamically generated schemas. It handles schema transitions efficiently when decoding, and administrators don’t have to worry about manually updating field names, tags, types, and so on, in order to make them compatible with new versions. Thrift and Protobuf, on the other hand, generate dynamic code and perform compile-time checks (type safety, and so forth) for different statically typed languages ​​such as C, C++, Java, C#, and more. Avro also supports dynamic code generation as optimization over statically typed languages. However, dynamic code generation is not required, especially when dealing with dynamically typed languages.

Advantages and Limitations#

Binary formats are much more compact and faster in terms of processing than text-based formats. They’re flexible with clearly defined version compatibility and dynamically-generated code support. However, when it comes to human interaction, they require preprocessing to make them human-readable. Let’s see how these binary formats map to our general criteria of an optimal data format, outlined in the table below:

Recommendations#

Choosing a data exchange format for an API is always debatable, because each format is suitable for some use cases and may have drawbacks for others. However, this section provides general recommendations for using the textual and binary data formats discussed in this chapter.

• JSON is probably the ideal choice when dealing with small groups of systems, especially those developed in JavaScript, where human-readability is essential.

• XML is probably the ideal format of choice when dealing with various systems with complex data structures that require markup and human-readability.

• Avro is likely the ideal choice when dealing with large files, frequent interactions of data encoded using different schema versions, and significant encoding sizes.

• Protobuf or Thrift may serve the purpose when network latency, interprocess communication, and processing speed are paramount.

The choice is not limited to the data formats discussed above; other formats, such as YAML, FormData, SQL, and so on, can also be considered. We can also introduce a custom format when an existing format doesn't meet our needs, keeping in mind the cost and effort required for internal or external usage. In other words, usage within the system or when sending information to end users.